Posterior image sampling using statistical learning model

ABSTRACT

Image reconstruction can include using a statistical or machine learning, MAP estimator, or other reconstruction technique to produce a reconstructed image from acquired imaging data. A Conditional Generative Adversarial Network (CGAN) technique can be used to train a Generator, using a Discriminator, to generate posterior distribution sampled images that can be displayed or further processed such as to help provide uncertainty information about a mean reconstruction image. Such uncertainty information can be useful to help understand or even visually modify the mean reconstruction image. Similar techniques can be used in a segmentation use-case, instead of a reconstruction use case. The uncertainty information can also be useful for other post-processing techniques.

CLAIM FOR PRIORITY

This application claims the benefit of priority to U.S. ProvisionalApplication Ser. No. 62/661,238, filed Apr. 23, 2018, which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This document pertains generally, but not by way of limitation, toposterior image sampling using a statistical learning model, including,but not by way of limitation, for use in planning or administration ofradiation treatment such as to a human or animal subject.

BACKGROUND

Radiation therapy can involve administering a dose of radiation to ahuman or animal subject. Careful planning can help ensure that theradiation reaches a target region of interest, while avoiding one ormore nearby regions that are not expected to benefit from radiation andthat may be impacted by side-effects of such radiation.

Three-dimensional (3D) imaging data can be used to characterize theinternal structure of a specimen, such as a human patient, such as tohelp plan radiation treatment in such a patient, such as to treat atumor. Such 3D imaging data can be obtained, for example, from amagnetic resonance (MR) or computed tomography (CT) imaging device orother imaging modality such as 3D electron microscopy. For example, the3D CT imaging data can include voxels representing imaging data ofvarious densities. For example, 3D CT voxel data of tissue within thesubject will represent a higher density than voxels representing airoutside of the subject. Voxels corresponding to air within a body cavity(e.g., within the bronchial tubes, for example, will also exhibit lessdensity than surrounding tissue. Bone tissue voxels will have a higherdensity than softer tissue voxels. In another example, the 3D MR imagingdata can include “k-space” values representing, in a spatial frequencydomain, MR imaging data information.

Image “reconstruction” is an example of an “inverse problem” ofgenerating structural information (e.g., an image of anatomicalstructures being studied) from acquired 3D CT imaging data or acquired3D MR imaging data. The imaging data can be (and usually is) noisy,which can make it difficult to accurately reconstruct an exact image ofthe interior of a patient. Image “segmentation” refers to partitioning areconstructed image into multiple regions, such as can permit locatingof boundaries between structures in the reconstructed image.

SUMMARY

The present inventors have recognized, among other things, that imagereconstruction can involve regularization of noisy acquired imaging datato produce a reconstructed image in which uncertainties in data can beinhibited from growing uncontrollably. For example, in CT imagereconstruction, a Filtered Back-Projection image reconstructiontechnique can be used. A more generic, yet adaptable approach, can bebased on reformulating the image reconstruction as an instance of Bayesinference, in which the goal is to recover the posterior distribution P(x|y), which represents the probability for the reconstructed image xgiven measured/acquired data y. The posterior distribution P (x|y)represents not only a single reconstructed image, it includesinformation about all possible reconstructed images, along withinformation about how likely such other possible reconstructed imagesare given the measured or acquired imaging data. Thus, the posteriordistribution P (x|y) can provide a reconstructed image along withuncertainty quantification.

In imaging applications, like those in biomedical and medical imaging,the posterior distribution P (x|y) can be highly dimensional, becauseimages can be described using a collection of many number values (e.g.,one number value per pixel or voxel). For example, a 3D image in conebeam computed tomography (CBCT) can result in a 10⁸ dimensional array.Hence, recovering the entire 3D posterior distribution P (x|y) can becomputationally intractable. An alternative to recovering the entire 3Dposterior distribution P (x|y) is to explore the posterior distributionby computing some estimator. For example, one can explore the posteriordistribution using a maximum a posterior (MAP) estimator, which can bedefined as:

$\underset{x}{\arg\;\max}{{P\left( x \middle| y \right)}.}$(Equation 1.) An iterative reconstruction approach can includeexplicitly or implicitly approximating a MAP estimator, givenappropriate choices for the underlying probabilities. Alternatively, amachine learning reconstruction approach can be used, such as can beshown to approximate a “conditional expectation” estimator of theposterior distribution, which can be defined as:∫xdP (x|y).(Equation 2.) Regardless of which estimator is used for reconstruction,in general, typical reconstruction approaches will yield a singlereconstructed image—which the present inventors have recognized canimply an enormous loss of important information. For example, all of theinformation contained in the posterior distribution P (x|y) related toquantifying the uncertainty associated with the reconstruction (usingthe estimator) may be lost by using only the single reconstructed image.

It is possible to quantify the point-wise (e.g., pixel-wise orvoxel-wise for a given or otherwise temporally-consistent time) standarddeviations in a given reconstructed image, but doing so would still onlycover a miniscule portion of the posterior distribution. Moreover, sucha point-wise approach misses any (possibly non-linear) causalitiesbetween image features. More precisely, point-wise standard deviationcan give point-wise bounds in the reconstructed image, but such anapproach cannot compute the probability of a tumor actually being thereor whether an antibody has bound to an antigen. The latter can involvemany pixels with possibly complicated casual relations that are notcaptured by only considering point-wise standard deviations in thereconstructed image.

The present inventors have recognized that a more complete analysis ofthe uncertainties can be obtained using a sampling from the posteriordistribution. Such an approach can allow one to query the probability ofthe likelihood of a certain image feature, but such an approach canpresent at least two disadvantages for biomedical and medical imagingapplications. First, it involves using a closed form of the posteriordistribution P(x|y), which is typically not available except in verysimplified cases. Second, sampling from the posterior distribution isgenerally based on designing structured random walks in high-dimensionalspaces, such as for example using Markov Chain Monte Carlo (MCMC)techniques, which can be very slow even when highly optimized. Thepresent techniques can help address both of these two potentialproblems.

The present techniques can include using one or more generative modelsin statistical or machine learning, such as can be combined with otherone or more other techniques in statistical or machine learning forsolving inverse problems (such as image reconstruction), such as tocreate a neural network that can allow quick or efficient sampling fromthe posterior distribution, such as can be useful to assess uncertaintyassociated with the reconstructed image.

As explained further herein, the present techniques can be implementedusing a generative model that can be conditioned on some input, e.g.,using a Conditional Variational Auto-Encoder (CVAE) or a ConditionalGenerative Adversarial Network (CGAN). For illustrative clarity, theexplanation given below in the Detailed Description of this patentdocument emphasizes a CGAN approach, however, the present techniques arenot so limited.

The above is intended to provide an overview of subject matter of thepresent patent application. It is not intended to provide an exclusiveor exhaustive explanation of the invention. The detailed description isincluded to provide further information about the present patentapplication.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. The drawings illustrate generally, by way of example, butnot by way of limitation, various embodiments discussed in the presentdocument.

FIG. 1 illustrates an example of a radiotherapy system.

FIG. 2 illustrates an example of a radiation therapy system that caninclude radiation therapy output configured to provide a therapy beam.

FIG. 3 illustrates an example of using a statistical or machine learningtechnique for sampling a posterior distribution such as for providinguncertainty information associated with a reconstructed image, such ascan be implemented at least in part using a device or machine such asshown in FIG. 6.

FIG. 4 illustrates the particular aspects of FIG. 3 that can be used, atrun-time, to provide uncertainty or error information about areconstructed image using the previously trained Generator.

FIGS. 5A, 5B, 5C, and 5D illustrate examples of how the posteriordistribution simulated images can be used to assess uncertaintyinformation associated with the mean reconstructed image.

FIG. 6 illustrates a block diagram of an embodiment of a device ormachine on which one or more of the methods as discussed herein can beimplemented, such as for using a statistical or machine learningtechnique for sampling a posterior distribution such as for providinguncertainty information associated with a reconstructed image.

DETAILED DESCRIPTION

Image reconstruction can include using a statistical or machinelearning, MAP estimator, or other reconstruction technique to produce areconstructed image from acquired imaging data. A Conditional GenerativeAdversarial Network (CGAN) technique can be used to train a Generator,using a Discriminator, to generate posterior distribution sampled imagesthat can be displayed or further processed such as to help provideuncertainty information about a mean reconstruction image. Suchuncertainty information can be useful to help understand or evenvisually modify the mean reconstruction image. Similar techniques can beused in a segmentation use-case, instead of a reconstruction use case.The uncertainty information can also be useful for other post-processingtechniques.

FIG. 1 illustrates an exemplary radiotherapy system 100 for providingradiation therapy to a patient, to a portion of a patient, or to a“phantom”, which can include a target object representing the patient orthe portion of the patient. The radiotherapy system 100 includes animage processing device, 112. The image processing device 112 may beconnected to a network 120. The network 120 may be connected to theInternet 122. The network 120 can connect the image processing device112 with one or more of a database 124, a hospital database 126, anoncology information system (OIS) 128, a radiation therapy device 130,an image acquisition device 132, a display device 134, and a userinterface 136. The image processing device 112 can be configured togenerate radiation therapy treatment plans 142 to be used by theradiation therapy device 130.

The image processing device 112 may include a memory device 116, aprocessor 114 and a communication interface 118. The memory device 116may store computer-executable instructions, such as an operating system143, a radiation therapy treatment plans 142 (e.g., original treatmentplans, adapted treatment plans and the like), software programs 144(e.g., artificial intelligence, deep learning, neural networks,radiotherapy treatment plan software), and any other computer-executableinstructions to be executed by the processor 114. In one embodiment, thesoftware programs 144 may convert medical images of one format (e.g.,MRI) to another format (e.g., CT) by producing synthetic images, such asa pseudo-CT image. For instance, the software programs 144 may includeimage processing programs to train a predictive model for converting amedial image 146 in one modality (e.g., an Mill image) into a syntheticimage of a different modality (e.g., a pseudo CT image); alternatively,the trained predictive model may convert a CT image into an MRI image.In another embodiment, the software programs 144 may register thepatient image (e.g., a CT image or an MR image) with that patient's dosedistribution (also represented as an image) so that corresponding imagevoxels and dose voxels are associated appropriately by the network. Inyet another embodiment, the software programs 144 may substitutefunctions of the patient images such as signed distance functions orprocessed versions of the images that emphasize some aspect of the imageinformation. Such functions might emphasize edges or differences invoxel textures, or any other structural aspect useful to neural networklearning. In another embodiment, the software programs 144 maysubstitute functions of the dose distribution that emphasize some aspectof the dose information. Such functions might emphasize steep gradientsaround the target, or any other structural aspect useful to neuralnetwork learning. The memory device 116 may store data, includingmedical images 146, patient data 145, and other data required to createand implement a radiation therapy treatment plan 142.

In addition to the memory 116 storing the software programs 144, it iscontemplated that software programs 144 may be stored on a removablecomputer medium, such as a hard drive, a computer disk, a CD-ROM, a DVD,a HD, a Blu-Ray DVD, USB flash drive, a SD card, a memory stick, or anyother suitable medium; and the software programs 144 when downloaded toimage processing device 112 may be executed by image processor 114.

The processor 114 may be communicatively coupled to the memory device116, and the processor 114 may be configured to execute computerexecutable instructions stored thereon. The processor 114 may send orreceive medical images 146 to memory 116. For example, the processor 114may receive medical images 146 from the image acquisition device 132 viathe communication interface 118 and network 120 to be stored in memory116. The processor 114 may also send medical images 146 stored in memory116 via the communication interface 118 to the network 120 be eitherstored in database 124 or the hospital database 126.

Further, the processor 114 may utilize software programs 144 (e.g., atreatment planning software) along with the medical images 146 andpatient data 145 to create the radiation therapy treatment plan 142.Medical images 146 may include information such as imaging dataassociated with a patient anatomical region, organ, or volume ofinterest segmentation data. Patient data 145 may include informationsuch as (1) functional organ modeling data (e.g., serial versus parallelorgans, appropriate dose response models, etc.); (2) radiation dosagedata (e.g., dose-volume histogram (DVH) information; or (3) otherclinical information about the patient and course of treatment (e.g.,other surgeries, chemotherapy, previous radiotherapy, etc.).

In addition, the processor 114 may utilize software programs to generateintermediate data such as updated parameters to be used, for example, bya neural network model; or generate intermediate 2D or 3D images, whichmay then subsequently be stored in memory 116. The processor 114 maysubsequently then transmit the executable radiation therapy treatmentplan 142 via the communication interface 118 to the network 120 to theradiation therapy device 130, where the radiation therapy plan will beused to treat a patient with radiation. In addition, the processor 114may execute software programs 144 to implement functions such as imageconversion, image segmentation, deep learning, neural networks, andartificial intelligence. For instance, the processor 114 may executesoftware programs 144 that train or contour a medical image; suchsoftware 144 when executed may train a boundary detector, or utilize ashape dictionary.

The processor 114 may be a processing device, include one or moregeneral-purpose processing devices such as a microprocessor, a centralprocessing unit (CPU), a graphics processing unit (GPU), an acceleratedprocessing unit (APU), or the like. More particularly, the processor 114may be a complex instruction set computing (CISC) microprocessor, areduced instruction set computing (RISC) microprocessor, a very longinstruction Word (VLIW) microprocessor, a processor implementing otherinstruction sets, or processors implementing a combination ofinstruction sets. The processor 114 may also be implemented by one ormore special-purpose processing devices such as an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA), adigital signal processor (DSP), a System on a Chip (SoC), or the like.As would be appreciated by those skilled in the art, in someembodiments, the processor 114 may be a special-purpose processor,rather than a general-purpose processor. The processor 114 may includeone or more known processing devices, such as a microprocessor from thePentium™, Core™, Xeon™, or Itanium® family manufactured by Intel™, theTurion™, Athlon™, Sempron™, Opteron™, FX™, Phenom™ family manufacturedby AMD™, or any of various processors manufactured by Sun Microsystems.The processor 114 may also include graphical processing units such as aGPU from the GeForce®, Quadro®, Tesla® family manufactured by Nvidia™,GMA, Iris™ family manufactured by Intel™, or the Radeon™ familymanufactured by AMD™. The processor 114 may also include acceleratedprocessing units such as the Xeon Phi™ family manufactured by Intel™.The disclosed embodiments are not limited to any type of processor(s)otherwise configured to meet the computing demands of identifying,analyzing, maintaining, generating, and/or providing large amounts ofdata or manipulating such data to perform the methods disclosed herein.In addition, the term “processor” may include more than one processor,for example, a multi-core design or a plurality of processors eachhaving a multi-core design. The processor 114 can execute sequences ofcomputer program instructions, stored in memory 116, to perform variousoperations, processes, methods that will be explained in greater detailbelow.

The memory device 116 can store medical images 146. In some embodiments,the medical images 146 may include one or more MRI image (e.g., 2D MRI,3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric MRI, 4D cine MRI, etc.),functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), ComputedTomography (CT) images (e.g., 2D CT, Cone beam CT, 3D CT, 4D CT),ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound),Positron Emission Tomography (PET) images, X-ray images, fluoroscopicimages, radiotherapy portal images, Single-Photo Emission ComputedTomography (SPECT) images, computer generated synthetic images (e.g.,pseudo-CT images) and the like. Further, the medical images 146 may alsoinclude medical image data, for instance, training images, and groundtruth images, contoured images, and dose images. In an embodiment, themedical images 146 may be received from the image acquisition device132. Accordingly, image acquisition device 132 may include a MRI imagingdevice, a CT imaging device, a PET imaging device, an ultrasound imagingdevice, a fluoroscopic device, a SPECT imaging device, an integratedLinear Accelerator and MRI imaging device, or other medical imagingdevices for obtaining the medical images of the patient. The medicalimages 146 may be received and stored in any type of data or any type offormat that the image processing device 112 may use to performoperations consistent with the disclosed embodiments. The memory device116 may be a non-transitory computer-readable medium, such as aread-only memory (ROM), a phase-change random access memory (PRAM), astatic random access memory (SRAM), a flash memory, a random accessmemory (RAM), a dynamic random access memory (DRAM) such as synchronousDRAM (SDRAM), an electrically erasable programmable read-only memory(EEPROM), a static memory (e.g., flash memory, flash disk, static randomaccess memory) as well as other types of random access memories, acache, a register, a compact disc read-only memory (CD-ROM), a digitalversatile disc (DVD) or other optical storage, a cassette tape, othermagnetic storage device, or any other non-transitory medium that may beused to store information including image, data, or computer executableinstructions (e.g., stored in any format) capable of being accessed bythe processor 114, or any other type of computer device. The computerprogram instructions can be accessed by the processor 114, read from theROM, or any other suitable memory location, and loaded into the RAM forexecution by the processor 114. For example, the memory 116 may storeone or more software applications. Software applications stored in thememory 116 may include, for example, an operating system 143 for commoncomputer systems as well as for software-controlled devices. Further,the memory 116 may store an entire software application, or only a partof a software application, that are executable by the processor 114. Forexample, the memory device 116 may store one or more radiation therapytreatment plans 142.

The image processing device 112 can communicate with the network 120 viathe communication interface 118, which can be communicatively coupled tothe processor 114 and the memory 116. The Communication interface 118may provide communication connections between the image processingdevice 112 and radiotherapy system 100 components (e.g., permitting theexchange of data with external devices). For instance, the communicationinterface 118 may in some embodiments have appropriate interfacingcircuitry to connect to the user interface 136, which may be a hardwarekeyboard, a keypad, or a touch screen through which a user may inputinformation into radiotherapy system 100.

Communication interface 118 may include, for example, a network adaptor,a cable connector, a serial connector, a USB connector, a parallelconnector, a high-speed data transmission adaptor (e.g., such as fiber,USB 3.0, thunderbolt, and the like), a wireless network adaptor (e.g.,such as a WiFi adaptor), a telecommunication adaptor (e.g., 3G, 4G/LTEand the like), and the like.

Communication interface 118 may include one or more digital and/oranalog communication devices that permit image processing device 112 tocommunicate with other machines and devices, such as remotely locatedcomponents, via the network 120.

The network 120 may provide the functionality of a local area network(LAN), a wireless network, a cloud computing environment (e.g., softwareas a service, platform as a service, infrastructure as a service, etc.),a client-server, a wide area network (WAN), and the like. For example,network 120 may be a LAN or a WAN that may include other systems S1(138), S2 (140), and S3 (141). Systems S1, S2, and S3 may be identicalto image processing device 112 or may be different systems. In someembodiments, one or more of systems in network 120 may form adistributed computing/simulation environment that collaborativelyperforms the embodiments described herein. In some embodiments, one ormore systems S1, S2, and S3 may include a CT scanner that obtain CTimages (e.g., medical images 146). In addition, network 120 may beconnected to internet 122 to communicate with servers and clients thatreside remotely on the internet.

Therefore, network 120 can allow data transmission between the imageprocessing device 112 and a number of various other systems and devices,such as the OIS 128, the radiation therapy device 130, and the imageacquisition device 132. Further, data generated by the OIS 128 and/orthe image acquisition device 132 may be stored in the memory 116, thedatabase 124, and/or the hospital database 126. The data may betransmitted/received via network 120, through communication interface118 in order to be accessed by the processor 114, as required.

The image processing device 112 may communicate with database 124through network 120 to send/receive a plurality of various types of datastored on database 124. For example, database 124 may include machinedata that is information associated with a radiation therapy device 130,image acquisition device 132, or other machines relevant toradiotherapy. Machine data information may include radiation beam size,arc placement, beam on and off time duration, machine parameters,segments, multi-leaf collimator (MLC) configuration, gantry speed, MRIpulse sequence, and the like. Database 124 may be a storage device andmay be equipped with appropriate database administration softwareprograms. One skilled in the art would appreciate that database 124 mayinclude a plurality of devices located either in a central or adistributed manner.

In some embodiments, database 124 may include a processor-readablestorage medium (not shown). While the processor-readable storage mediumin an embodiment may be a single medium, the term “processor-readablestorage medium” should be taken to include a single medium or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store the one or more sets of computerexecutable instructions or data. The term “processor-readable storagemedium” shall also be taken to include any medium that is capable ofstoring or encoding a set of instructions for execution by a processorand that cause the processor to perform any one or more of themethodologies of the present disclosure. The term “processor readablestorage medium” shall accordingly be taken to include, but not belimited to, solid-state memories, optical and magnetic media. Forexample, the processor readable storage medium can be one or morevolatile, non-transitory, or non-volatile tangible computer-readablemedia.

Image processor 114 may communicate with database 124 to read imagesinto memory 116 or store images from memory 116 to database 124. Forexample, the database 124 may be configured to store a plurality ofimages (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2DFluoroscopy images, X-ray images, raw data from MR scans or CT scans,Digital Imaging and Communications in Medicine (DIMCOM) data, etc.) thatthe database 124 received from image acquisition device 132. Database124 may store data to be used by the image processor 114 when executingsoftware program 144, or when creating radiation therapy treatment plans142. Database 124 may store the data produced by the trained neuralnetwork including the network parameters constituting the model learnedby the network and the resulting predicted data. The image processingdevice 112 may receive the imaging data 146 (e.g., 2D MRI slice images,CT images, 2D Fluoroscopy images, X-ray images, 3D MRI images, 4D MRIimages, etc.) either from the database 124, the radiation therapy device130 (e.g., a MRI-Linac), and or the image acquisition device 132 togenerate a treatment plan 142.

In an embodiment, the radiotherapy system 100 can include an imageacquisition device 132 that can acquire medical images (e.g., MagneticResonance Imaging (MRI) images, 3D MRI, 2D streaming MRI, 4D volumetricMRI, Computed Tomography (CT) images, Cone-Beam CT, Positron EmissionTomography (PET) images, functional MRI images (e.g., fMRI, DCE-MRI anddiffusion MRI), X-ray images, fluoroscopic image, ultrasound images,radiotherapy portal images, single-photo emission computed tomography(SPECT) images, and the like) of the patient. Image acquisition device132 may, for example, be an MRI imaging device, a CT imaging device, aPET imaging device, an ultrasound device, a fluoroscopic device, a SPECTimaging device, or any other suitable medical imaging device forobtaining one or more medical images of the patient. Images acquired bythe imaging acquisition device 132 can be stored within database 124 aseither imaging data and/or test data. By way of example, the imagesacquired by the imaging acquisition device 132 can be also stored by theimage processing device 112, as medical image data 146 in memory 116.

In an embodiment, for example, the image acquisition device 132 may beintegrated with the radiation therapy device 130 as a single apparatus(e.g., a MRI device combined with a linear accelerator, also referred toas an “MRI-Linac.” Such an MRI-Linac can be used, for example, todetermine a location of a target organ or a target tumor in the patient,so as to direct radiation therapy accurately according to the radiationtherapy treatment plan 142 to a predetermined target.

The image acquisition device 132 can be configured to acquire one ormore images of the patient's anatomy for a region of interest (e.g., atarget organ, a target tumor or both). Each image, typically a 2D imageor slice, can include one or more parameters (e.g., a 2D slicethickness, an orientation, and a location, etc.). In an embodiment, theimage acquisition device 132 can acquire a 2D slice in any orientation.For example, an orientation of the 2D slice can include a sagittalorientation, a coronal orientation, or an axial orientation. Theprocessor 114 can adjust one or more parameters, such as the thicknessand/or orientation of the 2D slice, to include the target organ and/ortarget tumor. In an embodiment, 2D slices can be determined frominformation such as a 3D MRI volume. Such 2D slices can be acquired bythe image acquisition device 132 in “near real-time” while a patient isundergoing radiation therapy treatment, for example, when using theradiation therapy device 130. “Near real-time” meaning acquiring thedata in at least milliseconds or less.

The image processing device 112 may generate and store radiation therapytreatment plans 142 for one or more patients. The radiation therapytreatment plans 142 may provide information about a particular radiationdose to be applied to each patient. The radiation therapy treatmentplans 142 may also include other radiotherapy information, such as beamangles, dose-histogram-volume information, the number of radiation beamsto be used during therapy, the dose per beam, and the like.

The image processor 114 may generate the radiation therapy treatmentplan 142 by using software programs 144 such as treatment planningsoftware, such as Monaco®, manufactured by Elekta AB of Stockholm,Sweden. In order to generate the radiation therapy treatment plans 142,the image processor 114 may communicate with the image acquisitiondevice 132 (e.g., a CT device, a MRI device, a PET device, an X-raydevice, an ultrasound device, etc.) to access images of the patient andto delineate a target, such as a tumor. In some embodiments, thedelineation of one or more organs at risk (OARs), such as healthy tissuesurrounding the tumor or in close proximity to the tumor may berequired. Therefore, segmentation of the OAR may be performed when theOAR is close to the target tumor. In addition, if the target tumor isclose to the OAR (e.g., prostate in near proximity to the bladder andrectum), then by segmenting the OAR from the tumor, the radiotherapysystem 100 may study the dose distribution not only in the target, butalso in the OAR.

In order to delineate a target organ or a target tumor from the OAR,medical images, such as MRI images, CT images, PET images, fMRI images,X-ray images, ultrasound images, radiotherapy portal images, SPECTimages and the like, of the patient undergoing radiotherapy may beobtained non-invasively by the image acquisition device 132 to revealthe internal structure of a body part. Based on the information from themedical images, a 3D structure of the relevant anatomical portion may beobtained. In addition, during a treatment planning process, manyparameters may be taken into consideration to achieve a balance betweenefficient treatment of the target tumor (e.g., such that the targettumor receives enough radiation dose for an effective therapy) and lowirradiation of the OAR(s) (e.g., the OAR(s) receives as low a radiationdose as possible). Other parameters that may be considered include thelocation of the target organ and the target tumor, the location of theOAR, and the movement of the target in relation to the OAR. For example,the 3D structure may be obtained by contouring the target or contouringthe OAR within each 2D layer or slice of an MRI or CT image andcombining the contour of each 2D layer or slice. The contour may begenerated manually (e.g., by a physician, dosimetrist, or health careworker using a program such as MONACO™ manufactured by Elekta AB ofStockholm, Sweden) or automatically (e.g., using a program such as theAtlas-based auto-segmentation software, ABAS™, manufactured by Elekta ABof Stockholm, Sweden). In certain embodiments, the 3D structure of atarget tumor or an OAR may be generated automatically by the treatmentplanning software.

After the target tumor and the OAR(s) have been located and delineated,a dosimetrist, physician or healthcare worker may determine a dose ofradiation to be applied to the target tumor, as well as any maximumamounts of dose that may be received by the OAR proximate to the tumor(e.g., left and right parotid, optic nerves, eyes, lens, inner ears,spinal cord, brain stem, and the like). After the radiation dose isdetermined for each anatomical structure (e.g., target tumor, OAR), aprocess known as inverse planning may be performed to determine one ormore treatment plan parameters that would achieve the desired radiationdose distribution. Examples of treatment plan parameters include volumedelineation parameters (e.g., which define target volumes, contoursensitive structures, etc.), margins around the target tumor and OARs,beam angle selection, collimator settings, and beam-on times. During theinverse-planning process, the physician may define dose constraintparameters that set bounds on how much radiation an OAR may receive(e.g., defining full dose to the tumor target and zero dose to any OAR;defining 95% of dose to the target tumor; defining that the spinal cord,brain stem, and optic structures receive ≤45Gy, ≤55Gy and <54Gy,respectively). The result of inverse planning may constitute a radiationtherapy treatment plan 142 that may be stored in memory 116 or database124. Some of these treatment parameters may be correlated. For example,tuning one parameter (e.g., weights for different objectives, such asincreasing the dose to the target tumor) in an attempt to change thetreatment plan may affect at least one other parameter, which in turnmay result in the development of a different treatment plan. Thus, theimage processing device 112 can generate a tailored radiation therapytreatment plan 142 having these parameters in order for the radiationtherapy device 130 to provide radiotherapy treatment to the patient.

In addition, the radiotherapy system 100 may include a display device134 and a user interface 136. The display device 134 may include one ormore display screens that display medical images, interface information,treatment planning parameters (e.g., contours, dosages, beam angles,etc.) treatment plans, a target, localizing a target and/or tracking atarget, or any related information to the user. The user interface 136may be a keyboard, a keypad, a touch screen or any type of device that auser may input information to radiotherapy system 100. Alternatively,the display device 134 and the user interface 136 may be integrated intoa device such as a tablet computer, e.g., Apple iPad®, Lenovo Thinkpad®,Samsung Galaxy®, etc.

Furthermore, any and all components of the radiotherapy system 100 maybe implemented as a virtual machine (e.g., VMWare, Hyper-V, and thelike). For instance, a virtual machine can be software that functions ashardware. Therefore, a virtual machine can include at least one or morevirtual processors, one or more virtual memories, and one or morevirtual communication interfaces that together function as hardware. Forexample, the image processing device 112, the OIS 128, the imageacquisition device 132 could be implemented as a virtual machine. Giventhe processing power, memory, and computational capability available,the entire radiotherapy system 100 could be implemented as a virtualmachine.

FIG. 2A illustrates an exemplary radiation therapy device 202 that mayinclude a radiation source, such as an X-ray source or a linearaccelerator, a couch 216, an imaging detector 214, and a radiationtherapy output 204. The radiation therapy device 202 may be configuredto emit a radiation beam 208 to provide therapy to a patient. Theradiation therapy output 204 can include one or more attenuators orcollimators, such as a multi-leaf collimator (MLC).

In FIG. 2, a patient can be positioned in a region 212, supported by thetreatment couch 216 to receive a radiation therapy dose according to aradiation therapy treatment plan. The radiation therapy output 204 canbe mounted or attached to a gantry 206 or other mechanical support. Oneor more chassis motors (not shown) may rotate the gantry 206 and theradiation therapy output 204 around couch 216 when the couch 216 isinserted into the treatment area. In an embodiment, gantry 206 may becontinuously rotatable around couch 216 when the couch 216 is insertedinto the treatment area. In another embodiment, gantry 206 may rotate toa predetermined position when the couch 216 is inserted into thetreatment area. For example, the gantry 206 can be configured to rotatethe therapy output 204 around an axis (“A”). Both the couch 216 and theradiation therapy output 204 can be independently moveable to otherpositions around the patient, such as moveable in transverse direction(“T”), moveable in a lateral direction (“L”), or as rotation about oneor more other axes, such as rotation about a transverse axis (indicatedas “R”). A controller communicatively connected to one or more actuators(not shown) may control the couch 216 movements or rotations in order toproperly position the patient in or out of the radiation beam 208according to a radiation therapy treatment plan. As both the couch 216and the gantry 206 are independently moveable from one another inmultiple degrees of freedom, which allows the patient to be positionedsuch that the radiation beam 208 precisely can target the tumor.

The coordinate system (including axes A, T, and L) shown in FIG. 2 canhave an origin located at an isocenter 210. The isocenter can be definedas a location where the central axis of the radiation therapy beam 208intersects the origin of a coordinate axis, such as to deliver aprescribed radiation dose to a location on or within a patient.Alternatively, the isocenter 210 can be defined as a location where thecentral axis of the radiation therapy beam 208 intersects the patientfor various rotational positions of the radiation therapy output 204 aspositioned by the gantry 206 around the axis A.

Gantry 206 may also have an attached imaging detector 214. The imagingdetector 214 preferably located opposite to the radiation source 204,and in an embodiment, the imaging detector 214 can be located within afield of the therapy beam 208.

The imaging detector 214 can be mounted on the gantry 206 preferablyopposite the radiation therapy output 204, such as to maintain alignmentwith the therapy beam 208. The imaging detector 214 rotating about therotational axis as the gantry 206 rotates. In an embodiment, the imagingdetector 214 can be a flat panel detector (e.g., a direct detector or ascintillator detector). In this manner, the imaging detector 214 can beused to monitor the therapy beam 208 or the imaging detector 214 can beused for imaging the patient's anatomy, such as portal imaging. Thecontrol circuitry of radiotherapy device 202 may be integrated withinsystem 100 or remote from it.

In an illustrative embodiment, one or more of the couch 216, the therapyoutput 204, or the gantry 206 can be automatically positioned, and thetherapy output 204 can establish the therapy beam 208 according to aspecified dose for a particular therapy delivery instance. A sequence oftherapy deliveries can be specified according to a radiation therapytreatment plan, such as using one or more different orientations orlocations of the gantry 206, couch 216, or therapy output 204. Thetherapy deliveries can occur sequentially, but can intersect in adesired therapy locus on or within the patient, such as at the isocenter210. A prescribed cumulative dose of radiation therapy can thereby bedelivered to the therapy locus while damage to tissue nearby the therapylocus can be reduced or avoided.

FIG. 2 illustrates generally illustrate an embodiment of a radiationtherapy device configured to provide radiotherapy treatment to apatient, including a configuration where a radiation therapy output canbe rotated around a central axis (e.g., an axis “A”). Other radiationtherapy output configurations can be used. For example, a radiationtherapy output can be mounted to a robotic arm or manipulator havingmultiple degrees of freedom. In yet another embodiment, the therapyoutput can be fixed, such as located in a region laterally separatedfrom the patient, and a platform supporting the patient can be used toalign a radiation therapy isocenter with a specified target locus withinthe patient.

In another embodiment, a radiation therapy device can be a combinationof a linear accelerator and an image acquisition device. In someembodiments, the image acquisition device may be an MRI, an X-ray, a CT,a CBCT, a spiral CT, a PET, a SPECT, an optical tomography, afluorescence imaging, ultrasound imaging, or radiotherapy portal imagingdevice, etc., as would be recognized by one of ordinary skill in theart.

Techniques to Apply a Statistical Learning Model for Posterior Samplingof an Image

The present inventors have recognized, among other things, a need tohelp improve image reconstruction or image segmentation, such as byusing a statistical or machine learning technique for sampling aposterior distribution such as for providing uncertainty informationassociated with a reconstructed image.

FIG. 3 illustrates an example of using a statistical or machine learningtechnique for sampling a posterior distribution such as for providinguncertainty information associated with a reconstructed image, such ascan be implemented using image processor circuitry or at least in partusing a device or machine such as shown in FIG. 6. FIG. 3 illustratesaspects of both training a statistical or machine learning model andusing such a trained model at run-time such as for sampling a posteriordistribution such as for providing uncertainty information associatedwith a reconstructed image. First model training will be described withrespect to FIG. 3, then use of the trained-model at run-time forproviding uncertainty information about a reconstructed image will bedescribed further with respect to FIG. 4.

At 300, measured or acquired imaging data of an object can be receivedor otherwise obtained. Such imaging data of the imaged object may havebeen previously acquired by an imaging modality (e.g., CT, MR, or otherimaging modality), and then stored, such as in memory circuitry that canbe included within or communicatively accessible by the image processorcircuitry.

At 302, an initial image reconstruction can be performed upon theimaging data obtained at 300. In an example, the initial imagereconstruction can use a technique that can yield an approximation tothe conditional mean (e.g., such as using an approach based onstatistical or machine learning, or by using a MAP estimator or thelike). This initial reconstructed image 304 can be referred to as an“initial image” or as “the mean image.”

At 304, the initial image can be fed to a Generator 306 convolutionalnetwork of a Conditional Generative Adversarial Network (CGAN) that alsoincludes a Discriminator 308 convolutional network. In this document,the term “convolutional” is understood to include the possibility of“deconvolutional.” For example, the Generator 306 and the Discriminator308 can form an adversarial network architecture in which theDiscriminator 308 can be adjusted during training such as to enhance ormaximize a value of an objective over a distribution of inputs to theDiscriminator 308, and the Generator 306 convolutional network can beadjusted during training to reduce or minimize the value of theobjective over a distribution of inputs to the Generator 306.

An independent noise source at 310 (e.g., normally distributed randomvalues) can be fed, together with the initial image at 304, to theGenerator 306. The Generator 306 can use a statistical learning modelrepresented by its convolutional network to combine the random noisevalues at 310 with the initial image at 304. The resulting output fromthe Generator 306 can include one or multiple posterior distributionsimulated images 312 (which can also be referred to as posteriordistribution sampled images, or even more simply as “samples” from theposterior distribution P(x|y)). These posterior distribution simulatedimages at 312 can be used at run-time to provide information about theuncertainty associated with the initial image 304, such as explainedfurther herein.

During CGAN training of the Generator 306, the Discriminator 308 alsoreceives the initial image at 304. During training, the Generatorgenerates, from the initial image at 304 and the random noise values at310, posterior distribution simulated images at 312, which isconditioned on the initial image at 304 provided to the Discriminator308. During training, these posterior distribution simulated images at312 can be provided to the Discriminator 308. Also during training, acorresponding deemed true image 314 (e.g., corresponding to the initialimage at 304) of the object is also provided to the Discriminator 308.During training, the Discriminator 308 uses its convolutional networkand the initial image at 304 to distinguish between the posteriordistribution simulated images at 312 and the deemed true image 314 fortraining a statistical learning model for use by the convolutionalnetwork of the Generator 306 in then later generating posteriordistribution simulated images at 312 at run-time (e.g., after training)such as for determining an image error or uncertainty associated with asubsequently obtained at least one reconstructed initial image at 304.During training, since both the Discriminator 308 and the Generator 306have access to the initial image at 304 (e.g., the mean image), the CGANtraining can be seen as conditioning on the image data 300. The deemedtrue image 314 can also be an image of the same object, that is,corresponding to the initial image 304 of the object, but with thedeemed true image 314 obtained with a higher degree of accuracy (lessuncertainty), such as by using more radiation, for example, in a CTimaging modality example.

FIG. 4 illustrates the particular aspects of FIG. 3 that can be used, atrun-time, to provide uncertainty or error information about areconstructed image using the previously trained Generator 306, withsuch training of the Generator 306 carried out such as described abovewith respect to FIG. 6. In the example of FIG. 4, the Discriminator 308need not be used at run-time. At run-time, a statistical learning, MAPestimator, or other reconstruction at 302 can be used to produce a meanimage 304 from previously-acquired image data. Uncertainty informationabout the mean image can be provided by the previously-trained Generator306, operating on the mean image at 304 in combination with random noisevalues at 310 to generate posterior distribution simulated images at312, which can provide uncertainty information associated with the meanimage 304, such as explained further herein.

FIGS. 5A, 5B, 5C, and 5D illustrate examples of how the posteriordistribution simulated images at 312 can be used to assess uncertaintyinformation associated with the mean image 304. FIG. 5A shows an exampleof a deemed true image 314 of an example of an object. FIG. 5B shows anexample of a corresponding mean reconstructed image 304, such as can beobtained by applying a statistical learning reconstruction technique, aMAP estimator reconstruction technique, or another reconstructiontechnique to acquired imaging data 300 of the object. FIG. 5C shows anexample of one of several posterior distribution simulated images at312, such as generated as described herein using the Generator 306 withthe mean reconstructed image 304 and the random noise values at 310 asinputs to the Generator 306. Each of the posterior distributionsimulated images at 312 provides an alternative representation of themean reconstruction image 304 shown in FIG. 5B, taking into account adegree of uncertainty associated with the introduction of the randomnoise values at 310 that were provided as inputs to the Generator 306.For example, by stepping through ones of the posterior distributionsimulated images at 312 (an example of one of which is shown in FIG.5C), a visual representation of the uncertainty associated with the meanreconstructed image 304 can help guide interpretation of the meanreconstructed image 304. For example, if the dark spot within theinterior of the deemed true image 314 in FIG. 5A represents a tumorwithin an object such as a human subject, the various posteriordistribution simulated images 312 in FIG. 5C can provide alternativerepresentations accounting for noise-induced uncertainties. This canallow the user to better visually interpret the mean reconstructionimage 304 shown in FIG. 5B, such as to assess whether there is a tumorfully enclosed within the interior of the image (such as shown in thetrue image 314 of FIG. 5A), when the mean reconstruction image 304 suchas shown in FIG. 5B is unclear or ambiguous with respect to whether sucha feature exists. The collection of posterior distribution simulatedimages 312 may provide at some samples 312 within the uncertaintydistribution that may more clearly hint at the existence of such afeature, even with the ambiguity in the mean reconstruction image 304shown in FIG. 5B.

Thus, using the trained architecture shown in FIG. 4 at run-time, it ispossible to quickly generate samples from the posterior distributionP(x|y), such as to provide posterior distribution simulated images 312.Such posterior distribution simulated images 312 can enable one to solvevarious statistical decision problems associated with the imagereconstruction problem, such as can permit using Bayesian hypothesistests to test for the visibility of a tumor, or to compute a point-wiseerror from the various posterior distribution simulated images 312, suchas can be depicted in a colored or grayscale or other shaded compositeimage (e.g., a “heat map”), such as depicted in FIG. 5D for 1000posterior distribution simulated images 312 shown in FIG. 5C. Performingcomputations such as shown in FIG. 5D would not be computationallyfeasible using a method such as a Markov Chain Monte Carlo (MCMC)technique.

Display of uncertainty information can include using a variety oftechniques. For example, the numerous posterior distribution sampledimages 312 can be displayed in temporal succession, such as in a videodisplay. Such a video can allow spatiotemporal evaluation of a candidatefeature, such as can include determining a spatiotemporal presence ofthe candidate feature in, across, or between the various image frames ofthe video. Such spatiotemporal evaluation can include viewing orotherwise using information about how often a candidate feature ispresent in the video, whether a spatiotemporal position of the candidatefeature is moving in the video images or is disappearing and reappearingspatiotemporally in the video, or the like. Such techniques can helpprovide valuable insight as to the nature of one or more featuresassociated with the mean reconstructed image such as shown in FIG. 5B.

Further, the uncertainty information can be used to modify the meanreconstructed image 304 such as shown in FIG. 5B. Since the meanreconstructed image 304 often represents a smoothed image (to accountfor noise), its display may be missing relevant “texture” that would bemeaningful to a user, but that was smoothed out by the reconstructiontechnique used to generate the mean reconstructed image 304 in FIG. 5B.The uncertainty information can be processed to generate and display avisual texture that is based on an indicator of the degree ofuncertainty, such as within one or more regions of the reconstructedimage, and the reconstructed image can be modified accordingly andpresented as an appropriately textured image for display to a user.

The pointwise error composite image shown in FIG. 5D is merely oneexample of a way to convey uncertainty information associated with amean reconstructed image 304 as shown in FIG. 5B. For example, insteadof displaying the uncertainty information such as shown in FIG. 5C or5D, the uncertainty information can be provided to another component ofthe image processor circuitry or other circuitry for furtherpost-processing. For example, such post-processing can include computingan aggregate error index over all or a specified portion of the image.In an example, such post-processing can include estimating a probabilityof existence of a particular feature, such as using information aboutthe spatiotemporal presence of the particular feature across some or allof the posterior distribution sampled images 312. Some otherillustrative non-limiting examples of further post-processing that canmake use of such uncertainty information can include, for example,segmentation or other point-wise (e.g., pixel-wise or voxel-wise) orother classification, radiation dose-computation, or the like. Forexample, radiation dose computation can use density information from areconstructed image—the available of point-wise or other uncertaintyinformation can be used along with the mean density information in theradiation dose-computation, such as can help provide more accuratelytailored radiation dose to a tumor or other desired delivery regionwithin a subject while helping better avoid delivering unwantedradiation to one or more organs at risk (OARs).

Although the above description has emphasized a use case of the presentposterior distribution sampling technique for exploring uncertainty witha mean reconstructed image use case, the techniques described herein canalso be used where segmentation is performed at 302 upon reconstructedimage data at 300 to produce a mean segmented image at 304, rather thanwhere reconstruction is performed at 302 upon acquired imaging data 300to produce a mean reconstructed image at 304, such as described andexplained above. In the segmentation use-case, the true image 314 is adeemed true segmented image. For example, the deemed true segmentedimage can be an image that has been segmented by one or more humanexperts (e.g., by one or several radiologists or other medical doctors,possibly with access to further information beyond the reconstructed orother image data (e.g., biopsy information, information from one or moreother imaging modalities, or other further information), or that hasbeen segmented by one or more already trained machines, possibly withaccess to further information.

As mentioned above, the present techniques can be implemented using agenerative model that can be conditioned on some input, e.g., using aConditional Variational Auto-Encoder (CVAE) or a Conditional GenerativeAdversarial Network (CGAN). For illustrative clarity, the explanationgiven below in the Detailed Description of this patent documentemphasizes a CGAN approach, however, the present techniques are not solimited. For example, aspects of the present techniques can be employedusing a Conditional Variational Auto-Encoder (CVAE) or a ConditionalGenerative Adversarial Network (CGAN) instead of a CGAN. Further,aspects of the present techniques can be used in a system employing aGAN, VAE, or other machine learning technique, without requiring a modelthat is conditioned upon some input, as would be the case with the CGANimplementation emphasized herein.

Deep Direct Estimation Approach

The above-described Deep Posterior Sampling approach for quantifyinguncertainty in image reconstruction can use generative models frommachine learning to create random samples s_(i) from the probabilitydistribution given by P(x=x|y=y). Using such generated random samples, awide range of one or more estimators can be evaluated. For example,according to the law of large numbers, the posterior mean can beapproximated according to

${{\mathbb{E}}\left\lbrack {\left. x \middle| y \right. = y} \right\rbrack} \approx {\frac{1}{n}{\sum\limits_{i = 1}^{n}s_{i}}}$Likewise, the posterior (pointwise) variance is given by

${{\mathbb{E}}\left\lbrack {\left. \left( {x - {{\mathbb{E}}\left\lbrack {\left. x \middle| y \right. = y} \right\rbrack}} \right)^{2} \middle| y \right. = y} \right\rbrack} \approx {\frac{1}{n}{\sum\limits_{j - 1}^{n}\;\left( {s_{j} - {\frac{1}{n}{\sum\limits_{i = 1}^{n}s_{i}}}} \right)^{2}}}$Such samples can also be used to answer one or more questions, such as,for example, “what is the probability of there being a tumor at aparticular location?” by checking how commonly the tumor is present inthe various generated samples.

The present inventors have recognized, among other things, thatclassical (MAP) estimators only reconstruct a single image, whichimplies an enormous loss of important information. As an example, all ofthe information contained in the posterior distribution P(x=x|y=y)related to quantifying the uncertainty associated with thereconstruction (estimator) is lost. The same problem applies to“classical” machine learning approaches that only give an approximationof the conditional mean.

While the above-described Deep Posterior Sampling approach can solveproblems such as these elegantly and in a computationally feasiblemanner, the computational runtime can still be non-negligible. Forexample, generating each sample si can take about 10 milliseconds perslice, e.g., using hardware currently available in 2018. To accuratelycompute an estimator, hundreds of samples can be needed, which cantranslate to run-times on the order of seconds per slice, or minutes fora 3D volume.

Alternatively or additionally, a deep direct estimation approach caninclude using one or more machine learning approaches to directlycompute one or more predetermined estimators, such as of the formdescribed below, such as without requiring any sampling. For example,consider any estimator of the form

[f(x,y)|y=y]

In the above, f: X×Y→Z is an arbitrary function. For example, aconditional mean can be obtained using f(x, y)=x and the conditional(pointwise) variance by using f(x, y)=(x−

[x|y=y])².

In a practical example, the probability distribution of data may not beknown. Instead, access to training data (x_(i), y_(i)) is only availablefrom the joint distribution P(x=x, y=y). Hence, computing theconditional expectation in the above is impossible, since we don't havesufficient training data from the conditional distribution P(x=x|y=y).This means that we cannot compute the above type of estimators directly.

To solve this problem, the present inventors have recognized thefollowing statement, which can be proven.

[f(x,y)|y=⋅]=argmin_(R:Y→X)

∥R(y)−f(x,y)∥_(Z) ²

Note that in the above statement, to the left there exists a conditionalexpectation, while on the right we simply have the standard expectation.Thus, our training data does not allow directly approximation of theleft-hand side, but such training data can be used to evaluate theright-hand side.

The present inventors have recognized that, among other things, thismeans that any estimator that can be written as a conditional estimatorcan be characterized as the function that can be used to solve anoptimization problem over the joint distribution. While the usefulnessof this by itself may be limited, the function R:Y→X can be replacedwith a parametrized function R_(θ) where θ can represent the parameters.Specifically of interest are neural networks, such as in which θ willinclude the parameters (weights, biases, etc) of the network. Further,the expected value to the right can be replaced with our training data.Thus, it can be shown that solving the learning problem,

$\theta^{*} = {\arg\;{\min_{\theta}{\frac{1}{n}{\sum\limits_{i = 1}^{n}{{{R_{\theta}\left( y_{i} \right)} - {f\left( {x_{i},y_{i}} \right)}}}_{Z}^{2}}}}}$which is to find the best weights for our training data, yields a directestimatorR _(θ*)(y)≈

[f(x,y)|y=y]To put this is less mathematical terms, it can often be desired tocompute one or more estimators that can be given as a conditionalexpectation of some function. Such estimators can be very useful incertain applications, e.g., the conditional mean and variance are ofhigh interest. It is also possible to compute even more intricateestimators, such as the average bone density, the outline (e.g.,segmentation) of a tumour, or the like. Doing so directly may not befeasible, such as where there is not enough training data to compute theconditional expectation. Instead, the present inventors have recognizedthat it is possible to re-write the estimator as the minimizer of anoptimization problem with a very specific choice of objective function.This optimization problem is of the same type as is typically found indeep learning applications. Thus, it can be well-solved using a deeplearning approach.

After training (which can be done off-line) we can obtain a functionR_(θ*):Y→Z that is given by a neural network. To evaluate the estimator,we only need to evaluate this neural network once. This is very fast,taking only ≈10 milliseconds per slice, or seconds for a 3D volume.

The method as given above is general, e.g., it can be applied to a verywide range of settings, including tomographic imaging, but also to otherthings, such as image segmentation, radiation dose computation, etc.

ILLUSTRATIVE NUMBERED EXAMPLES OF DEEP DIRECT ESTIMATION Example 1

(Training) A method of training a neural network for processing imagingor other spatially distributed data using processor circuitry applying astatistical learning model, the method comprising:

obtaining imaging or other spatially distributed training data;

selecting an estimator of interest that is capable of being representedas a conditional expectation, the estimator of interest capable of beingrepresented in a parameterized form by a trained neural network;

training the neural network by solving an optimization problem over ajoint distribution, wherein the optimization problem has the estimatorof interest as a minimizer.

Example 2

(Runtime) A method of processing imaging or other spatially distributeddata using processor circuitry applying a statistical learning model,the method comprising:

obtaining imaging or other spatially distributed data corresponding to apatient of interest;

using a trained neural network and the imaging or other spatiallydistributed data for performing at least one of image reconstruction,image segmentation, or radiation dose computation; and

providing the results of the at least one of image reconstruction, imagesegmentation, or radiation dose computation for at least one ofdisplaying or further processing; and

wherein the neural network is previously trained according to Example 1.

Example 3

The method of Example 2, comprising using the trained neural network andthe imaging or other spatially distributed data for performinguncertainty estimation for at least one of image reconstruction, imagesegmentation, or radiation dose computation.

FIG. 6 illustrates a block diagram of an embodiment of a device ormachine 1000 on which one or more of the methods as discussed herein canbe implemented. One or more items of the image processing device 112 canbe implemented by the machine 1000. The machine 1000 can operate as astandalone device or may be connected (e.g., networked) to othermachines. The image processing device 112 can include one or more of theitems of the machine 1000. In a networked deployment, the machine 1000may operate in the capacity of a server or a client machine inserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine may be apersonal computer (PC), a tablet PC, a set-top box (STB), a PersonalDigital Assistant (PDA), a cellular telephone, a web appliance, anetwork router, switch or bridge, or any machine capable of executinginstructions (sequential or otherwise) that specify actions to be takenby that machine. Further, while only a single machine is illustrated,the term “machine” shall also be taken to include any collection ofmachines that individually or jointly execute a set (or multiple sets)of instructions to perform any one or more of the methodologiesdiscussed herein.

The example machine 1000 can include processing circuitry 1002 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU), anapplication specific integrated circuit, circuitry, such as one or moretransistors, resistors, capacitors, inductors, diodes, logic gates,multiplexers, buffers, modulators, demodulators, radios (e.g., transmitor receive radios or transceivers), sensors 1021 (e.g., a transducerthat converts one form of energy (e.g., light, heat, electrical,mechanical, or other energy) to another form of energy), or the like, ora combination thereof), a main memory 1004 and a static memory 1006,which communicate with each other via a bus 1008. A datum or dataassociated with the described methods can be stored in or retrieved fromsuch memory, and initialized or updated as desired to carry out themethods described herein. The machine 1000 (e.g., computer system) mayfurther include a video display unit 1010 (e.g., a liquid crystaldisplay (LCD) or a cathode ray tube (CRT)). The machine 1000 can alsoinclude an alphanumeric input device 1012 (e.g., a keyboard), a userinterface (UI) navigation device 1014 (e.g., a mouse), a disk drive ormass storage unit 1016, a signal generation device 1018 (e.g., aspeaker) and a network interface device 1020.

The disk drive unit 1016 can include a machine-readable medium 1022 onwhich is stored one or more sets of instructions and data structures(e.g., software) 1024 embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1024 mayalso reside, completely or at least partially, within the main memory1004 and/or within the processor 1002 during execution thereof by themachine 1000, the main memory 1004 and the processor 1002 alsoconstituting machine-readable media.

The machine 1000 as illustrated can include an output controller 1028.The output controller 1028 manages data flow to/from the machine 1000.The output controller 1028 can sometimes be called a device controller,with software that directly interacts with the output controller 1028being called a device driver.

While the machine-readable medium 1022 is shown in an embodiment to be asingle medium, the term “machine-readable medium” may include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or moreinstructions or data structures. The term “machine-readable medium”shall also be taken to include any tangible medium that is capable ofstoring, encoding or carrying instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present invention, or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including by way of example semiconductormemory devices, e.g., Erasable Programmable Read-Only Memory (EPROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 1024 may further be transmitted or received over acommunications network 1026 using a transmission medium. Theinstructions 1024 may be transmitted using the network interface device1020 and any one of a number of well-known transfer protocols (e.g.,HTTP). Examples of communication networks include a local area network(“LAN”), a wide area network (“WAN”), the Internet, mobile telephonenetworks, Plain Old Telephone (POTS) networks, and wireless datanetworks (e.g., WiFi and WiMax networks). The term “transmission medium”shall be taken to include any intangible medium that is capable ofstoring, encoding or carrying instructions for execution by the machine,and includes digital or analog communications signals or otherintangible media to facilitate communication of such software.

As used herein, “communicatively coupled between” means that theentities on either of the coupling must communicate through an itemtherebetween and that those entities cannot communicate with each otherwithout communicating through the item.

The above description includes references to the accompanying drawings,which form a part of the detailed description. The drawings show, by wayof illustration, specific embodiments in which the invention can bepracticed. These embodiments are also referred to herein as “examples.”Such examples can include elements in addition to those shown ordescribed. However, the present inventors also contemplate examples inwhich only those elements shown or described are provided. Moreover, thepresent inventors also contemplate examples using any combination orpermutation of those elements shown or described (or one or more aspectsthereof), either with respect to a particular example (or one or moreaspects thereof), or with respect to other examples (or one or moreaspects thereof) shown or described herein.

In the event of inconsistent usages between this document and anydocuments so incorporated by reference, the usage in this documentcontrols.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of “at least one” or “one or more.” In thisdocument, the term “or” is used to refer to a nonexclusive or, such that“A or B” includes “A but not B,” “B but not A,” and “A and B,” unlessotherwise indicated. In this document, the terms “including” and “inwhich” are used as the plain-English equivalents of the respective terms“comprising” and “wherein.” Also, in the following claims, the terms“including” and “comprising” are open-ended, that is, a system, device,article, composition, formulation, or process that includes elements inaddition to those listed after such a term in a claim are still deemedto fall within the scope of that claim. Moreover, in the followingclaims, the terms “first,” “second,” and “third,” etc. are used merelyas labels, and are not intended to impose numerical requirements ontheir objects.

Geometric terms, such as “parallel”, “perpendicular”, “round”, or“square”, are not intended to require absolute mathematical precision,unless the context indicates otherwise. Instead, such geometric termsallow for variations due to manufacturing or equivalent functions. Forexample, if an element is described as “round” or “generally round,” acomponent that is not precisely circular (e.g., one that is slightlyoblong or is a many-sided polygon) is still encompassed by thisdescription.

Method examples described herein can be machine or computer-implementedat least in part. Some examples can include a computer-readable mediumor machine-readable medium encoded with instructions operable toconfigure an electronic device to perform methods as described in theabove examples. An implementation of such methods can include code, suchas microcode, assembly language code, a higher-level language code, orthe like. Such code can include computer readable instructions forperforming various methods. The code may form portions of computerprogram products. Further, in an example, the code can be tangiblystored on one or more volatile, non-transitory, or non-volatile tangiblecomputer-readable media, such as during execution or at other times.Examples of these tangible computer-readable media can include, but arenot limited to, hard disks, removable magnetic disks, removable opticaldisks (e.g., compact disks and digital video disks), magnetic cassettes,memory cards or sticks, random access memories (RAMs), read onlymemories (ROMs), and the like.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments can be used, such as by one of ordinary skill in the artupon reviewing the above description. The Abstract is provided to complywith 37 C.F.R. § 1.72(b), to allow the reader to quickly ascertain thenature of the technical disclosure. It is submitted with theunderstanding that it will not be used to interpret or limit the scopeor meaning of the claims. Also, in the above Detailed Description,various features may be grouped together to streamline the disclosure.This should not be interpreted as intending that an unclaimed disclosedfeature is essential to any claim. Rather, inventive subject matter maylie in less than all features of a particular disclosed embodiment.Thus, the following claims are hereby incorporated into the DetailedDescription as examples or embodiments, with each claim standing on itsown as a separate embodiment, and it is contemplated that suchembodiments can be combined with each other in various combinations orpermutations. The scope of the invention should be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

The claimed invention is:
 1. A method of image processing using imageprocessor circuitry applying a trained statistical learning model, themethod comprising: obtaining imaging data of a human subject acquired atleast in part using a medical imaging modality; applying at least one ofa reconstruction or a segmentation to the imaging data to generate atleast one reconstructed or segmented initial image; providing theinitial image and an image randomness component to a generator neuralnetwork, the generator neural network applying a previously trainedconditional generative statistical learning model; generating, with thegenerator neural network, a plurality of posterior distributionsimulated images from the initial image and the image randomnesscomponent; analyzing the posterior distribution simulated images, toidentify at least one indication of an image error associated with theinitial image; and providing the indication of the image errorassociated with the initial image for displaying or further processing.2. The method of claim 1, wherein providing the indication of the imageerror associated with the initial image includes providing a point-wiseindication of the image error associated with the initial image.
 3. Themethod of claim 2, wherein providing the point-wise indication of theimage error associated with the initial image includes displaying theposterior distribution simulated images as frames in a video.
 4. Themethod of claim 2, wherein providing the point-wise indication of theimage error associated with the initial image includes displaying acomposite error image determined from the posterior distributionsimulated images.
 5. The method of claim 1, comprising, using the imageprocessor circuitry, evaluating a feature at least in part using theposterior distribution simulated images.
 6. The method of claim 5,wherein evaluating the feature includes determining how often thefeature is present in the posterior distribution simulated images. 7.The method of claim 5, wherein evaluating the feature includesdetermining whether a spatiotemporal position of the feature is changingbetween the posterior distribution simulated images.
 8. The method ofclaim 5, wherein evaluating the feature includes determining aspatiotemporal presence of the feature in the posterior distributionsimulated images.
 9. The method of claim 1, comprising modifying theinitial image using the indication of the image error for display to auser.
 10. The method of claim 9, wherein the initial image includes amean reconstruction image, and wherein the modifying the initial imageincludes adding a variation to texture at least one region of the meanreconstruction image.
 11. The method of claim 1, wherein the applying atleast one of a reconstruction or a segmentation to the imaging data togenerate at least one reconstructed or segmented initial image includesapplying a reconstruction to the imaging data to generate at least onereconstructed initial image.
 12. The method of claim 1, wherein applyinga reconstruction or a segmentation to the imaging data to generate atleast one reconstructed or segmented initial image includes applying asegmentation to the imaging data to generate at least one segmentedinitial image.
 13. The method of claim 1 wherein the generator neuralnetwork applies a statistical learning model trained including by:providing at least one reconstructed or segmented initial image of anobject to a discriminator network and a generator network of aconditional generative adversarial network (GAN); providing at least onecorresponding deemed true image of the object to the discriminatornetwork; using the generator network to generate, from the initial imageand from an image randomness component, a posterior distributionsimulated image, conditioned on the initial image provided to thediscriminator network; and using the discriminator network to use theinitial image to distinguish between the posterior distributionsimulated images and the deemed true image to train thestatistical-learning model to be applied by the generator network. 14.The method of claim 1, wherein presenting the indication of the imageerror includes displaying on an MR-LINAC workstation for use inradiation treatment planning or administration.
 15. The method of claim1, comprising updating, or receiving an update, of a radiation treatmentprotocol based at least in part upon the indication of the image error.16. The method of claim 1, wherein the generator neural networkimplements a first convolutional neural network of a conditionalgenerative adversarial network (GAN) that also includes a discriminatorneural network that implements a second convolutional neural network.17. The method of claim 1, wherein the generator neural networkimplements a variational auto-encoder (VAE).
 18. A method of trainingimage processor circuitry using statistical learning, the methodcomprising: providing at least one reconstructed or segmented initialimage of an anatomical object captured from a human subject to adiscriminator neural network and a generator neural network; providingat least one corresponding deemed true image of the anatomical object tothe discriminator neural network; using the generator neural network togenerate, from the initial image and an image randomness component,posterior distribution simulated images, conditioned on the initialimage, provided to the discriminator neural network; and using thediscriminator neural network and the initial image to distinguishbetween the posterior distribution simulated images and the deemed trueimage, causing training of a statistical learning model, the statisticallearning model adapted to be implemented by the generator neural networkin generative operations for generating posterior distribution simulatedimages; wherein analysis of the posterior distribution simulated imagesgenerated from a subsequently obtained at least one reconstructed orsegmented initial image enables identification of an image errorassociated with the subsequently obtained at least one reconstructed orsegmented initial image.
 19. The method of claim 18, comprising trainingthe discriminator neural network using the posterior distributionsimulated images and the deemed true image.
 20. The method of claim 19,comprising training the discriminator neural network using a conditionalgenerative adversarial network (CGAN).